Skip to content

Run tests and rule validation in CI#3

Merged
NicholasEllul merged 1 commit intoellul/initfrom
ellul/ci-checks
Jan 20, 2025
Merged

Run tests and rule validation in CI#3
NicholasEllul merged 1 commit intoellul/initfrom
ellul/ci-checks

Conversation

@NicholasEllul
Copy link
Contributor

@NicholasEllul NicholasEllul commented Jan 17, 2025 •
edited
Loading

Summary

This pull request adds a GitHub workflow that will run Semgrep rule validation and Semgrep tests in CI. This will help ensure the rules we are developing are vetted prior to merge.

Testing

  1. Make an adjustment to one of the existing rules in the rules/src/ directory that breaks the config file in some way (e.g rename pattern to patern)
  2. Run bin/validate-rules and see that it gets flagged.
Example output 
/src/metamask/semgrep-action (ellul/ci-checks) » bin/validate-rules                         
Configuration is invalid - found 2 configuration error(s), and 0 rule(s).
semgrep error: Invalid rule schema
  --> rules/src/github-actions/publish-actions-cache-used.yml:11
11 |       - patterns:
12 |           - patern: "uses: $ACTION_NAME"
13 |           - metavariable-regex:
14 |               metavariable: $ACTION_NAME
15 |               regex: actions/cache@v[0-9]+
16 |     paths:

Additional properties are not allowed ('patterns' was unexpected)

[ERROR] Rule parse error in rule publish-actions-cache-used:
 unexpected key patern
  1. Correct the rule you broke above, and modify one of the files in the rules/test directory such that one of the lines flagging a rule (which look like --> ruleid: publish-actions-cache-used), is renamed to # ok: publish-actions-cache-used.
  2. See that Semgrep warns you that a test did not pass.
Example output 
~/src/metamask/semgrep-action (ellul/ci-checks*) » bin/test                             
0/1: 1 unit tests did not pass:
--------------------------------------------------------------------------------
        ✖ publish-actions-cache-used
        missed lines: [], incorrect lines: [89]
        test file path: /Users/ellul/src/metamask/semgrep-action/rules/test/github-actions/publish-actions-cache-used.test.yaml


No tests for fixes found.

@NicholasEllul NicholasEllul force-pushed the ellul/ci-checks branch 4 times, most recently from 98a754a to 563073b Compare January 17, 2025 17:16
@NicholasEllul NicholasEllul force-pushed the ellul/add-actions-cache-rule branch from 165f3ae to be89eb7 Compare January 17, 2025 17:20
Base automatically changed from ellul/add-actions-cache-rule to ellul/init January 17, 2025 17:32
@NicholasEllul NicholasEllul merged commit ed61475 into ellul/init Jan 20, 2025
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@witmicko witmicko witmicko approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@NicholasEllul @witmicko